Contactless Monitoring of Respiratory Rate and Breathing Absence from Head Movements Using an RGB Camera.

Contactless vital sign monitoring is more demanding for long-term, continuous, and unobtrusive measurements. Camera-based respiratory monitoring is receiving growing interest with advanced video technologies and computational power. The volume variations of the lungs for airflow changes create a periodic movement of the torso, but identifying the torso is more challenging than face detection in a video. In this paper, we present a unique approach to monitoring respiratory rate (RR) and breathing absence by leveraging head movements alone from an RGB video because respiratory motion also influences the head. Besides our novel RR estimation, an independent algorithm for breathing absence detection using signal feature extraction and machine learning techniques identifies an apnea event and improves overall RR estimation accuracy. The proposed approach was evaluated using videos from 30 healthy subjects who performed various breathing tasks. The breathing absence detector had 0.87 F1 score, 0.9 sensitivity, and 0.85 specificity. The accuracy of spontaneous breathing rate estimation increased from 2.46 to 1.91 bpm MAE and 3.54 to 2.7 bpm RMSE when combining the breathing absence result with the estimated RR.Clinical relevance- Our contactless respiratory monitoring can utilize a consumer RGB camera to offer a significant benefit in continuous monitoring of neonatal monitoring, sleep monitoring, telemedicine or telehealth, home fitness with mild physical movement, and emotion detection in the clinic and remote locations.

Enhanced Contactless Heart Rate Monitoring Using Camera with Motion Artifact Removal During Physical Activities.

Contactless monitoring of heart rate (HR) can improve passive and continuous tracking of cardiovascular activities and overall people's health. Remote photoplethysmography (rPPG) using a camera eliminates the need for a wearable device. rPPG-based HR has shown promising results to be accurate and comparable to conventional methods such as contact PPG. Most experiments use stationary subjects while motion is known to affect the accuracy of remote PPG. In this paper, a novel methodology is introduced to enhance the accuracy and reliability of HR monitoring based on rPPG in the presence of physical activities like Yoga. This method quickly and accurately tracks HR and analyzes head motion to exclude unreliable data within short windows of rPPG signals. The method was tested with smartphone video data collected from 60 subjects when they are doing activities with varying levels of movement. Results show that our method without motion removal improves the accuracy of the HR readings by 0.7 bpm, reaching 3.57 bpm on average for a 30-sec-window. The accuracy is further improved by another 1.3 bpm after removing the motion artifacts, and reaches 2.29 bpm.Clinical relevance- The enhancement of HR readings from shorter rPPG signal with motion tolerance during physical activities can ultimately help with a more reliable HR tracking of people in uncontrolled settings like home which is a critical step towards remote health-care or wellness tracking.

Estimating SpO2 with Deep Oxygen Desaturations from Facial Video Under Various Lighting Conditions: A Feasibility Study.

This paper presents a feasibility study to collect data, process signals, and validate accuracy of peripheral oxygen saturation (SpO2) estimation from facial video in various lighting conditions. We collected facial videos using RGB camera, without auto-tuning, from subjects when they were breathing through a mouth tube with their nose clipped. The videos were record under four lighting conditions: warm color temperature and normal brightness, neutral color temperature and normal brightness, cool color temperature and normal brightness, neutral color temperature and dim brightness. The air inhaled by the subjects was manually controlled to gradually induce hypoxemia and lower subjects' SpO2 to as low as 81%. We first extracted the remote photoplethysmogram (rPPG) signals from the videos. We applied the principle of pulse oximetry and extracted the ratio of ratios (RoR) for two color combinations: Red/Blue and Red/Green. Next, we assessed SpO2 estimation accuracy against the ground truth, a Transfer Standard Pulse Oximeter. We have achieved an RMSE of 1.93% and a PCC of 0.97 under the warm color temperature and normal brightness lighting condition using leave-one-subject-out cross validation between two subjects. The results have demonstrated the feasibility to estimate SpO2 remotely and accurately using consumer level RGB camera with suitable camera configuration and lighting condition.Clinical Relevance- This work demonstrates that SpO2 can be estimated accurately using an RGB camera without auto-tuning and under warm color temperature, enabling continuous SpO2 monitoring applications that require noncontact sensing.

Utilizing Deep Learning on Limited Mobile Speech Recordings for Detection of Obstructive Pulmonary Disease.

Passive assessment of obstructive pulmonary disease has gained substantial interest over the past few years in the mobile and wearable computing communities. One of the promising approaches is speech-based pulmonary assessment wherein spontaneous or scripted speech is used to evaluate an individual's pulmonary condition. Recent approaches in this regard heavily rely on accurate speech activity segmentation and specific, hand-crafted features. In this paper, we present an end-to-end deep learning approach for detecting obstructive pulmonary disease. We leveraged transfer learning using a network pre-trained for a different audio-based task, and employed our own additional shallow network on top as a binary classifier to indicate if a given speech recording belongs to an asthma or COPD patient. The additional network was a fully connected neural net with 2 hidden layers, and this was evaluated on two real-world datasets. We demonstrated that the system can identify subjects with obtructive pulmonary disease using their speech with 88.3 % precision, 88.8 % recall and 88.3% F-1 score using 10-fold cross-validation. The system showed improved performance in identifying the most severely affected subgroup of patients in the dataset, with an average 93.6 % accuracy.

Deep Multivariate Domain Translation for Device Invariant Pulmonary Patient Identification from Cough and Speech Sounds.

Pulmonary audio sensing from cough and speech sounds in commodity mobile and wearable devices is increasingly used for remote pulmonary patient monitoring, home healthcare, and automated disease analysis. Patient identification is important for such applications to ensure system accuracy and integrity, and thus avoiding errors and misdiagnosis. Widespread usage and deployment of such patient identification models across various devices are challenging due to domain shift of acoustic features because of device heterogeneity. Because of this phenomenon, a patient identification model developed using audio data collected with one type of device is not usable when deployed in another type of device, which is a concern for model portability and general usability. This paper presents a framework utilizing a multivariate deep neural network regressor as a feature translator between source device and target device domains to reduce the effect of domain shift for better model portability. Extensive and empirical experiments of our translation framework consisting of two different human sound (speech and cough) based pulmonary patient identification tasks using audio data collected from 91 real patients demonstrate that it can recover up to 64.8% of lost accuracy due to domain shift across two common and widely used mobile and wearable devices: smartphone and smartwatch. Clinical Relevance- The methods presented in this paper will enable efficient and easy portability of pulmonary patient identification models from cough and speech across various mobile and wearable devices used by a patient.

Remote Photoplethysmography and Heart Rate Estimation by Dynamic Region of Interest Tracking.

Remote photoplethysmography (PPG) estimates vital signs by measuring changes in the reflected light from the human skin. Compared to traditional PPG techniques, remote PPG enables contactless measurement at a reduced cost. In this paper, we propose a novel method to extract remote PPG signals and heart rate from videos. We propose an algorithm to dynamically track regions of interest (ROIs) and combine the signals from all ROIs based on signal qualities. To maintain a stable frame rate and accuracy, we propose a dynamic down-sampling approach, which makes our system robust to the different video resolutions and user-camera distances. We also propose the strategy of adaptive measurement time to estimate HR, which can achieve comparable accuracy in HR estimation while reducing the average measurement time. To test the accuracy of the proposed system, we have collected data from 30 subjects with facial masks. Experimental results show that the proposed system can achieve 3.0 bpm mean absolute error in HR estimation.

Motion- Based Respiratory Rate Estimation with Motion Artifact Removal Using Video of Face and Upper Body.

Respiratory rate (RR) is a significant indicator of health conditions. Remote contactless measurement of RR is gaining popularity with recent respiratory tract infection awareness. Among various methods of contactless RR measurement, a video of an individual can be used to obtain an instantaneous RR. In this paper, we introduce an RR estimation based on the subtle motion of the head or upper chest captured on an RGB camera. Motion-based respiratory monitoring allows us to acquire RR from individuals with partial face coverings, such as glasses or a face mask. However, motion-based RR estimation is vulnerable to the subject's voluntary movement. In this work, adaptive selection between face and chest regions plus a motion artifact removal technique enables us to obtain a much cleaner respiratory signal from the video recordings. The average mean absolute error (MAE) for controlled and natural breathing is 1.95 BPM using head motion only and 1.28 BPM using chest motion only. Our results demonstrate the possibility of continuous monitoring of breathing rate in real-time with any personal device equipped with an RGB camera, such as a laptop or a smartphone.

SpeechSpiro: Lung Function Assessment from Speech Pattern as an Alternative to Spirometry for Mobile Health Tracking.

Respiratory illnesses are common in the United States and globally; people deal with these illnesses in various forms, such as asthma, chronic obstructive pulmonary diseases, or infectious respiratory diseases (e.g., coronavirus). The lung function of subjects affected by these illnesses degrades due to infection or inflammation in their respiratory airways. Typically, lung function is assessed using in-clinic medical equipment, and quite recently, via portable spirometry devices. Research has shown that the obstruction and restriction in the respiratory airways affect individuals' voice characteristics. Hence, audio features could play a role in predicting the lung function and severity of the obstruction. In this paper, we go beyond well-known voice audio features and create a hybrid deep learning model using CNN-LSTM to discover spatiotemporal patterns in speech and predict the lung function parameters with accuracy comparable to conventional devices. We validate the performance and generalizability of our method using the data collected from 201 subjects enrolled in two studies internally and in collaboration with a pulmonary hospital. SpeechSpiro measures lung function parameters (e.g., forced vital capacity) with a mean normalized RMSE of 12% and R2 score of up to 76% using 60-second phone audio recordings of individuals reading a passage.Clinical relevance - Speech-based spirometry has the potential to eliminate the need for an additional device to carry out the lung function assessment outside clinical settings; hence, it can enable continuous and mobile track of the individual's condition, healthy or with a respiratory illness, using a smartphone.

A Comprehensive Approach for Classification of the Cough Type.

Identifying the presence of sputum in the lung is essential in detection of diseases such as lung infection, pneumonia and cancer. Cough type classification (dry/wet) is an effective way of examining presence of lung sputum. This is traditionally done through physical exam in a clinical visit which is subjective and inaccurate. This work proposes an objective approach relying on the acoustic features of the cough sound. A total number of 5971 coughs (5242 dry and 729 wet) were collected from 131 subjects using Smartphone. The data was reviewed and annotated by a novel multi-layer labeling platform. The annotation kappa inter-rater agreement score is measured to be 0.81 and 0.37 for 1st and 2nd layer respectively. Sensitivity and specificity values of 88% and 86% are measured for classification between wet and dry coughs (highest across the literature).

CoughGAN: Generating Synthetic Coughs that Improve Respiratory Disease Classification.

Despite the prevalence of respiratory diseases, their diagnosis by clinicians is challenging. Accurately assessing airway sounds requires extensive clinical training and equipment that may not be easily available. Current methods that automate this diagnosis are hindered by their use of features that require pulmonary function tests. We leverage the audio characteristics of coughs to create classifiers that can distinguish common respiratory diseases in adults. Moreover, we build on recent advances in generative adversarial networks to augment our dataset with cleverly engineered synthetic cough samples for each class of major respiratory disease, to balance and increase our dataset size. We experimented on cough samples collected with a smartphone from 45 subjects in a clinic. Our CoughGAN-improved Support Vector Machine and Random Forest models show up to 76% test accuracy and 83% F1 score in classifying subjects' conditions between healthy and three major respiratory diseases. Adding our synthetic coughs improves the performance we can obtain from a relatively small unbalanced healthcare dataset by boosting the accuracy over 30%. Our data augmentation reduces overfitting and discourages the prediction of a single, dominant class. These results highlight the feasibility of automatic, cough-based respiratory disease diagnosis using smartphones or wearables in the wild.

CoughMatch - Subject Verification Using Cough for Personal Passive Health Monitoring.

Automatic cough detection using audio has advanced passive health monitoring on devices such as smart phones and wearables; it enables capturing longitudinal health data by eliminating user interaction and effort. One major issue arises when coughs from surrounding people are also detected; capturing false coughs leads to significant false alarms, excessive cough frequency, and thereby misdiagnosis of user condition. To address this limitation, in this paper, a method is proposed that creates a personal cough model of the primary subject using limited number of cough samples; the model is used by the automatic cough detection to verify whether the identified coughs match the personal pattern and belong to the primary subject. A Gaussian mixture model is trained using audio features from cough to implement the subject verification method; novel cough embeddings are learned using neural networks and integrated into the model to further improve the prediction accuracy. We analyze the performance of the method using our cough dataset collected by a smart phone in a clinical study. Population in the dataset involves subjects categorized of healthy or patients with COPD or Asthma, with the purpose of covering a wider range of pulmonary conditions. Cross-subject validation on a diverse dataset shows that the method achieves an average error rate of less than 10%, using a personal cough model generated by only 5 coughs from the primary subject.

Adapting to Noise in Speech Obfuscation by Audio Profiling Using Generative Models for Passive Health Monitoring.

Passive health monitoring has been introduced as a solution for continuous diagnosis and tracking of subjects' condition with minimal effort. This is partially achieved by the technology of passive audio recording although it poses major audio privacy issues for subjects. Existing methods are limited to controlled recording environments and their prediction is significantly influenced by background noises. Meanwhile, they are too compute-intensive to be continuously running on smart phones. In this paper, we implement an efficient and robust audio privacy preserving method that profiles the background audio to focus only on audio activities detected during recording for performance improvement, and to adapt to the noise for more accurate speech segmentation. We analyze the performance of our method using audio data collected by a smart watch in lab noisy settings. Our obfuscation results show a low false positive rate of 20% with a 92% true positive rate by adapting to the recording noise level. We also reduced model memory footprint and execution time of the method on a smart phone by 75% and 62% to enable continuous speech obfuscation.

Toward Early Severity Assessment of Obstructive Lung Disease Using Multi-Modal Wearable Sensor Data Fusion During Walking.

Early detection of chronic diseases helps to minimize the disease impact on patient's health and reduce the economic burden. Continuous monitoring of such diseases helps in the evaluation of rehabilitation program effectiveness as well as in the detection of exacerbation. The use of everyday wearables i.e. chest band, smartwatch and smart band equipped with good quality sensor and light weight machine learning algorithm for the early detection of diseases is very promising and holds tremendous potential as they are widely used. In this study, we have investigated the use of acceleration, electrocardiogram, and respiration sensor data from a chest band for the evaluation of obstructive lung disease severity. Recursive feature elimination technique has been used to identity top 15 features from a set of 62 features including gait characteristics, respiration pattern and heart rate variability. A precision of 0.93, recall of 0.91 and F-1 score of 0.92 have been achieved with a support vector machine for the classification of severe patients from the non-severe patients in a data set of 60 patients. In addition, the selected features showed significant correlation with the percentage of predicted FEV1.Clinical Relevance- The study result indicates that wearable sensor data collected during natural walk can be used in the early evaluation of pulmonary patients thus enabling them to seek medical attention and avoid exacerbation. In addition, it may serve as a complementary tool for pulmonary patient evaluation during a 6-minute walk test.

Estimation of the Lung Function Using Acoustic Features of the Voluntary Cough.

Spirometry test, a measure of the patient's lung function, is the gold standard for diagnosis and monitoring of chronic pulmonary diseases. Spirometry is currently being done in hospital settings by having the patients blow the air out of their lungs forcefully and into the spirometer's tubes under the supervision and constant guidance of clinicians. This test is expensive, cumbersome and not easily applicable to every-day monitoring of these patients. The lung mechanism when performing a cough is very similar to when spirometry test is done. That includes a big inhalation, air compression and forceful exhalation. Therefore, it is reasonable to assume that obstruction of lung airways should have a similar effect on both cough features and spirometry measures. This paper explores the estimation of lung obstruction using cough acoustic features. A total number of 3695 coughs were collected from patients from 4 different conditions and 4 different severity categories along with their lung function measures in a clinical setting using a smartphone's microphone and a hospital-grade spirometry lab. After feature-set optimization and model hyperparameter tuning, the lung obstruction was estimated with MAE (Mean Absolute Error) of 8% for COPD and 9% for asthma populations. In addition to lung obstruction estimation, we were able to classify patients' disease state with 91% accuracy and patients' severity within each disease state with 95% accuracy.Clinical Relevance- This enables effort-independent estimation of lung function spirometry parameters which could potentially lead to passive monitoring of pulmonary patients.